The disclosures of the following priority applications are herein incorporated by reference: Japanese Patent Application No. 9-261359, filed Sep. 26, 1997 and Japanese Patent Application No. 9-293912, filed Oct. 27, 1997.
1. Field of Invention
The present invention relates to an interpolation device and process that perform interpolation of an image in which grid points that output a predetermined color and empty grid points that do not output the color are arrayed in a checkered configuration, and to a recording medium in which an interpolation processing program for use with such an image is recorded.
2. Description of Related Art
There are electronic cameras that use a single imaging element to produce an image signal of a color image. An imaging element like this, for example, as shown in FIG. 16A, is structured with color filters of the three colors of RGB (red color, green color, blue color) arranged in a Bayer array.
Referring to the green color (G), it can be said that the imaging element is structured as an array having some grid points, which are pixels that output green color as shown in FIG. 16B, and other empty grid points, which are pixels that do not output green color. These grid points are arrayed in a checkered configuration.
Furthermore, when using a color filter having a structure in which two imaging elements comprising only green color (G) as shown in FIG. 17A are shifted relative to each other, as shown in FIG. 17B, it can be said that imaginary empty grid points that do not output green color, and grid points that output green color are structured so as to be arrayed in a checkered configuration.
Furthermore, when using a two-plate-type of color filter, one imaging element is formed by G only, and the other imaging element is structured such that pixels that output red color and pixels that output blue color are arrayed in a checkered configuration. That is, the pixels that output blue color become empty grid points with respect to red color, and the pixels that output red color become empty grid points with respect to blue color.
Conventionally, in methods that perform interpolation of the color image that corresponds to the image signals formed by such imaging elements, the average value of the image (pixel) output of the grid points adjacent to empty grid points is used as the pixel output (interpolation amount) of the empty grid points.
U.S. Pat. No. 5,373,322 discloses a process that estimates the directionality of the image output of the empty grid points by using chrominance gradients as a technique for interpolating parts of an image, such as edges, having a spatially strong directivity.
FIG. 18 illustrates a conventional interpolation process disclosed in U.S. Pat. No. 5,373,322. In FIG. 18, G11, G13, . . . , G66, and so forth, show the pixel output of green color (G), R12, R14, . . . , R56, and so forth, show the pixel output of red color (R), and B21, B23, . . . , B67, and so forth, show the pixel output of blue color (B). In U.S. Pat. No. 5,373,322, as described above, when the interpolation amount G34 of green color of the pixel that outputs R34 is calculated, the information HDiff related to the chrominance gradients in the horizontal direction is obtained as
HDiff=|((R32+R36)/2)xe2x88x92R34|
and the information VDiff related to the chrominance gradients in the vertical direction is obtained as
VDiff=|((R14+R54)/2)xe2x88x92R34|.
The calculation method of G34 varies from case to case, based upon the relationship between HDiff and VDiff, which were thus obtained. That is, in the case of HDiff less than VDiff (when the correlation in the horizontal direction is strong), G34 is obtained by
G34=(G33+G35)/2
and in the case of VDiff less than HDiff (when the correlation in the vertical direction is strong), G34 is obtained by
G34=(G24+G44)/2
and in the case of HDiff=VDiff, G34 is obtained by
G34=(G24+G44+G33+G35)/4.
That is, the information related to the chrominance gradients of green color is estimated in this prior art technique based upon the pixel output of red color irrespective of the process that obtains the interpolation amount of green color.
Estimating the spatial similarity of the empty grid points in order to accurately predict the interpolation amount of the empty grid points is effective. However, the information related to the chrominance gradients of green color is estimated in the prior art based upon the pixel output of red color irrespective of the process that obtains the interpolation amount of green color. Because of this, it was not possible to accurately estimate the directivity of the pixel output of green color based upon the pixel output of red color in an image in which the chrominance gradients of green color and the chrominance gradients of red color are not constant, or in an image in which the component of red color is low, or the like.
Furthermnore, the directivity of the pixel output of the empty grid points is not always symmetrically right and left or symmetrically up and down. Because of this, for reasons similar to the prior art described above, when the information related to the chrominance gradients was obtained in only the horizontal and vertical directions, it was difficult to accurately estimate the directivity of the appropriate pixel output.
For example, when each pixel output in FIG. 18 has the value shown in FIG. 19, G34 has a strong correlation in the vertical direction (particularly in the upper direction).
U.S. Pat. No. 5,373,322 uses the following calculations:
HDiff=|((R32+R36)/2)xe2x88x92R34|
VDiff=|((R14+R54)/2)xe2x88x92R34|
In this example, HDiff=VDiff is established, and it is estimated that the correlation in the vertical and horizontal directions is the same. In such a situation, where HDiff=VDiff=0.5, it is preferable for G34=1 to be satisfied. However, in this situation, since HDiff=VDiff:   "AutoLeftMatch"                              G34          =                                    (                              G24                +                G44                +                G33                +                G35                            )                        /            4                                                        =          0.75                    
That is, G34=0.75, rather than G34=1. Thus, when the interpolation is performed based upon the directivity of the pixel output as conventionally estimated, there is a possibility of damaging the clearness of the image, or destroying the image, in the worst case. Because of this, there is a possibility that the sharpness and the smoothness of the border parts can be lost.
Thus, one object of the invention is to provide an interpolation device and process that estimates the spatial directivity of empty grid points in detail, and accurately obtains the interpolation amount of the empty grid points.
Another object of the invention is to provide a recording medium on which is recorded an interpolation processing program that estimates the spatial directivity of the empty grid points in detail and that accurately obtains the interpolation amount of the empty grid points.
An interpolation device according to one aspect of the invention performs interpolation of an image having grid points that output a predetermined color and at least one empty grid point that does not output the predetermined color, the grid points being oriented in a checkered configuration, and obtains an interpolation amount that becomes a pixel output of the empty grid point. A correlation amount calculator determines a correlation between individual pixel outputs of at least two grid points adjacent to the empty grid point and pixel outputs of each one of grid points oriented in shifted positions in the four directions of upper, lower, right, and left directions, respectively, with respect to the at least two grid points, and calculates a correlation amount showing the correlation strength in each direction. A similar direction detector identifies a direction having a strong correlation amount based upon the correlation amount for each direction calculated by the correlation amount calculator. An interpolation amount calculator determines an interpolation amount of the empty grid point based upon a pixel output of at least one grid point in a direction where correlation discerned by the similar direction detector is strong.
An interpolation device according to another aspect of the invention performs interpolation of an image having grid points that output a predetermined color and at least one empty grid point that does not output the predetermined color, the grid points being oriented in a checkered configuration, and obtains an interpolation amount that becomes a pixel output of the empty grid point. A correlation amount calculator determines a correlation between individual pixel outputs of at least two grid points adjacent to the empty grid point and pixel outputs of each one of grid points oriented in shifted positions in the four directions of upper, lower, right, and left directions, respectively, with respect to the at least two grid points. A weighting factor calculator calculates a weighting factor for each direction based upon the correlation amount for each direction calculated by the correlation amount calculator. An interpolation amount calculator calculates the product of the weighting factor for each direction calculated by the weighting factor calculator and the pixel outputs of grid points corresponding to each weighting factor, and assigns a sum of those products as an interpolation amount of the empty grid point.
According to another aspect of the invention, an interpolation device includes a first similarity calculator that calculates a first vertical direction similarity indicative of a similarity of two grid points that are adjacent in a vertical direction to an empty grid point and a first horizontal direction similarity indicative of a similarity of two grid points that are adjacent in a horizontal direction to the empty grid point. A second similarity calculator calculates a second vertical direction similarity indicative of a similarity of at least one of the grid points that are adjacent in a horizontal direction to the empty grid point and a second horizontal direction similarity indicative of a similarity of at least one of the grid points that are adjacent in a vertical direction to the empty grid point. A similarity detector calculates a similarity in a vertical direction of the empty grid point based on the first vertical direction similarity and the second vertical direction similarity and determines a similarity in a horizontal direction of the empty grid point based on the first horizontal direction similarity and the second horizontal direction similarity. An interpolation calculator calculates an interpolation amount of the empty grid point by selecting at least one pixel output of the grid points that are positioned in a direction where the similarity is strong according to the similarity determined by the similarity detector or by weighted addition of at least one pixel output of the grid points that are positioned in the vertical direction and at least one pixel output of the grid point positioned in the horizontal direction, according to the similarity.
The invention also relates to the processes performed by the interpolation device and to a recording medium, such as, for example, a CD-ROM or a carrier wave, that stores a computer readable program or data signal representative of the program to perform the interpolation process.